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DETAILED ACTION 

I. Continued Examination Under 37 CFR 1.114 

A request for continued examination under 37 CFR 1.114, including the 
fee set forth in 37 CFR 1.17(e), was filed in this application after final 
rejection. Since this application is eligible for continued examination 
under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been 
timely paid, the finality of the previous Office action has been withdrawn 
pursuant to 37 CFR 1.1 14. Applicant's submission filed on 25 November 
2008 has been entered. 

II. Response to Arguments 

Applicants' response to the last Office Action, mailed 30 July 2008 has 
been entered and made of record. 

Claims 1-3, 5-11, 13-28, and 32-41 are pending in the application. 

Applicant's amendments to independent claims 1, 11, 22, 32, 36, and 38 
require new grounds of rejection. New grounds of rejection are provided 
in the Office Action below. 

Applicant's arguments with respect to claims 1, 11, 22, 32, 36, and 38 
have been considered but are moot in view of the new grounds of 
rejection. 
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III. Claim Rejections - 35 USC § 101 

35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or 
composition of matter, or any new and useful improvement thereof, may obtain a patent 
therefor, subject to the conditions and requirements of this title. 

A. Claims 1-3, 5-11, and 13-28 are rejected under 35 U.S.C. 101 because the 
claimed invention is directed to non-statutory subject matter. 

Claims 1-3, 5-11, and 13-28 are rejected under 35 U.S.C. 101 as not 
falling within one of the four statutory categories of invention. Supreme 
Court precedent 1 and recent Federal Circuit decisions 2 indicate that a 
statutory "process" under 35 U.S.C. 101 must (1) be tied to another 
statutory category (such as a particular apparatus), or (2) transform 
underlying subject matter (such as an article or material) to a different 
state or thing. While the instant claims 1, 11, and 22 recite a series of 
steps or acts to be performed, the claims neither transform underlying 
subject matter nor positively tie to another statutory category that 
accomplishes the claimed method steps, and therefore do not qualify as a 
statutory process. For example, though the preambles of independent 
claims 1, 11, and 22 recite a "computerized" method, the steps of these 
claims do not explicitly or inherently require any system element in order 



1 Diamond v. Diehr, 450 U.S. 175, 184 (1981); Parker v. Flook, 437 U.S. 584, 588 n.9 (1978); Gottschalk v. 
Benson, 409 U.S. 63, 70 (1972); Cochrane v. Deener, 94 U.S. 780, 787-88 (1876). 

2 In reBilski, 88 USPQ2d 1385 (Fed. Cir. 2008). 
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to be implemented. At least one method step of each independent claim 
must require some computer element performing the step to make these 
claims statutory. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S. C. 103(a) which forms the basis for 
all obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or 
described as set forth in section 102 of this title, if the differences between the subject matter 
sought to be patented and the prior art are such that the subject matter as a whole would 
have been obvious at the time the invention was made to a person having ordinary skill in the 
art to which said subject matter pertains. Patentability shall not be negatived by the manner 
in which the invention was made. 

A. Claims 22-28, 32-35, 38, and 40-41 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Szeliski et al. (US Patent Number 6,157,747, herein 
referred to as Szeliski) in view of Seago (US Patent Number 5,990,900). 

Regarding independent claim 22, Szeliski teaches a computerized method 
for projecting texture information onto an object within an image 
panorama (See Szeliski; Fig. 2B, Item 270; Col. 27, Lns. 62-66), the 
method comprising: 

• Receiving instructions from a user identifying a three-dimensional 
geometric surface within an image panorama (See Szeliski: Col. 27-28, 
Lns. 62-2, respectively, wherein, "The shape of the model and the 
embedding of each face into texture space are left up to the user. This 
choice can range from something as simple as a cube with six 
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separate texture maps, to something as complicated as a subdivided 

dodecahedron, or even a latitude-longitude tessellated globe.", 

o The image panorama containing an object having one or more 

textures (See Szeliski: Col. 5, Lns. 27-35), 
o The object occupying a field of view of more than 180 degrees in 
the panorama (See Szeliski: Col. 5, Lns. 7-9, "By taking as many 
images as needed, image mosaics can be constructed which cover 
as large a field of view as desired, e.g., a complete environment 
map."); 

• Determining a directional vector from the three-dimensional geometric 
surface (See Szeliski: Fig. 15; Col. 22, Lns. 4-31); 

• Creating a geometric model of the image panorama based at least in 
part on the three-dimensional geometric surface and the directional 
vector (See Szeliski: Col. 7, Lns. 29-33, "By mapping the mosaic onto 
an arbitrary texture-mapped polyhedron surrounding the origin, the 
virtual environment is viewed using standard 3D graphics viewers and 
hardware (block 160) without requiring special purpose players." In 
Col. 22, Lns. 4-31, Szeliski teaches the direction vectors, which are 
used to align the images of the panoramic image. Thus, the geometric 
model of the visual scene is dependent on the geometric shapes of the 
surfaces of the polyhedron to be texture mapped, and the texture map 
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applied to the surfaces of the polyhedron, which is the panoramic 
mosaic determined using the direction vectors.); and 

• Applying the one or more textures to the object in the image 
panorama based on the geometric model (See Szeliski: Fig. 2B, Item 
270; Col. 28, Lns. 13-15, wherein Szeliski teaches, "...efficiently 
computing texture map color values for any geometry and choice of 
texture map coordinates."). 

Szeliski does not expressly suggest: 

• Creating a geometric model includes identifying at least one boundary 
of the object and 

• Using the identified boundary to associate geometry information with 
the object, 

o The geometry information comprising 3-D coordinates describing 
the position and orientation of the object boundary in a reference 
coordinate system. 
Seago is cited for teaching, "Line creation can be performed automatically 
by image analysis software, which determines edges of objects and 
creates lines overlapping the determined edges (See Seago: Col. 4, Line 
65 - Col. 5, Line 1, respectively)," which correspond to identifying at least 
one boundary of the object. Seago also teaches, "Next, at blocks 50, 
shapes or polygons that define the selected object's sides, and plane 
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indexes are determined based on user designated vertices or vanishing 
lines at significant features of the selected object (See Seago: Fig. 2, Item 
50; Col. 5, Lns. 35-38)," which corresponds to using the boundary to 
associate geometry information with the object. Finally, Seago is cited for 
teaching, "Once all the polygons and plane indexes of a selected object 
have been determined, a three-dimensional object oriented within the 
selected object's three-dimensional coordinate space is determined (See 
Seago: Fig. 2, Item 54; Col. 5, Lns. 50-55)," which corresponds to the 3D 
coordinates describing the position and orientation of the object 
boundary in the reference coordinate system. 

It would have been obvious to one of ordinary skill in the art, at the time 
the invention was made, to have used the three-dimensional object 
editing abilities of Seago, to modify the three-dimensional environment 
map, as taught by Szeliski, because Seago is in the same field of 
endeavor of generating three-dimensional computer graphics using image 
capture devices; and Seago expressly suggests that this method of object 
extraction produces accurate three-dimensional objects more efficiently 
than conventional systems that try to extract the three-dimensional 
objects using analytical mathematical interpretations and orthogonal 
image analysis (See Seago: Col. 11, Lns. 49-55). 
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Regarding independent claim 32, the Szeliski teaches a system for 
creating a three dimensional model from a plurality of image panoramas, 
the system comprising: 

• Means for receiving the image panoramas representing a visual scene 
(See Szeliski: Col. 27, Lns. 34-37, "Once a complete panoramic mosaic 
has been constructed, it is necessary to convert the set of input 
images and associated transforms into one or more images which can 
be quickly rendered or viewed.") having an object (See Szeliski: Fig. 6, 
Item 610; Col. 11, Lns. 16-21); 

o The object occupying a field of view of more than 180 degrees in 
the panoramas (See Szeliski: Col. 5, Lns. 7-9, "By taking as many 
images as needed, image mosaics can be constructed which cover 
as large a field of view as desired, e.g., a complete environment 
map."), 

• Means for allowing a user to interact with the system to determine a 
directional vector for each image panorama (See Szeliski: Col. 25, Lns. 
64-66); 

• Means for aligning the image panoramas relative to each other (See 
Szeliski: Fig. 17; Col. 23, Lns. 10-12, wherein minimizing the 
differences between ray directions aligns the images.). 

Szeliski does not expressly suggest: 
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o Means for creating a three dimensional model from the aligned 
panoramas, wherein: 

o Creating a three dimensional model includes identifying at least 

one boundary of the object and 
o Using the identified boundary to associate geometry information 

with the object includes associating geometry information with the 

object, 

■ The geometry information comprising 3-D coordinates 
describing the position and orientation of the object boundary 
in a reference coordinate system. 

Seago is cited for teaching: 

o Means for creating a three dimensional model from the aligned 
panoramas (See Seago: Fig. 2; Col. 4, Lns. 23-25, "FIG. 2 illustrates 
the process by which the image converting system 20 generates a 
three-dimensional model or object from a single two-dimensional 
image."), wherein: 

o Creating a three dimensional model includes identifying at least 
one boundary of the object (See Seago: Col. 4, Line 65 - Col. 5, Line 
1, respectively, "Line creation can be performed automatically by 
image analysis software, which determines edges of objects and 
creates lines overlapping the determined edges.") and 
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o Using the identified boundary to associate geometry information 
with the object includes associating geometry information with the 
object (See Seago: Fig. 2, Item 50; Col. 5, Lns. 35-38, "Next, at 
blocks 50, shapes or polygons that define the selected object's 
sides, and plane indexes are determined based on user designated 
vertices or vanishing lines at significant features of the selected 
object."), 

■ The geometry information comprising 3-D coordinates 
describing the position and orientation of the object boundary 
in a reference coordinate system (See Seago: Fig. 2, Item 54; 
Col. 5, Lns. 50-55, "Once all the polygons and plane indexes of 
a selected object have been determined, a three-dimensional 
object oriented within the selected object's three-dimensional 
coordinate space is determined."). 

It would have been obvious to one of ordinary skill in the art, at the time 
the invention was made, to have used the three-dimensional object 
editing abilities of Seago, to modify the three-dimensional environment 
map, as taught by Error! Reference source not found., because Seago 
is in the same field of endeavor of generating three-dimensional 
computer graphics using image capture devices; and Seago expressly 
suggests that this method of object extraction produces accurate three- 
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dimensional objects more efficiently than conventional systems that try 
to extract the three-dimensional objects using analytical mathematical 
interpretations and orthogonal image analysis (See Seago: Col. 11, Lns. 
49-55). 

Independent claim 38 differs from independent claim 32 primarily in that 
claim 38 is directed toward the computerized method implemented on 
the system of claim 32. Claim 38 also differs from claim 32 in that it 
requires creating a three dimensional model of the visual scene from the 
image panorama using the reference coordinate system. Seago is cited for 
teaching, "Next, at block 42, an object contained within the digital image 
is selected for conversion into a three- dimensional object, and the 
selected object's orientation, or natural coordinate system, is 
approximately determined (See Seago: Fig. 2, Item 42; Col. 4, Lns. 46-49, 
wherein the reference coordinate system is the natural coordinate 
system.)." Therefore, the rationale of claim 32 is applied to claim 38. 

Regarding claim 24, as it depends from claim 22, Szeliski teaches the 
three-dimensional geometric surface is one of a floor, a wall, or a ceiling 
(See Szeliski: Col. 27, Lns. 64-67, wherein the model can be a cube with 
six separate texture maps for each surface. Using an appropriate 
environment map would cause the top surface to be a ceiling, the bottom 
surface to be a floor, etc.). 
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Regarding claim 25, as it depends from claim 22, Szeliski teaches the 
directional vector is orthogonal to the planar surface (See Szeliski: Fig. 
15; Col. 22, Lns. 4-6). 

Regarding claim 26, as it depends from claim 22, Szeliski teaches the 
geometric model comprises depth information (See Szeliski: Figs. 27 and 
30; Col. 28, Lns. 29-33). 

Regarding claim 27, as it depends from claim 22, Szeliski teaches the 
texture information comprises color information (See Szeliski: Col. 28, 
Lns. 13-18). 

Regarding claim 28, as it depends from claim 22, Szeliski teaches the 
texture information comprises luminance information (See Szeliski: Fig. 
2B; Col. 5, Lns. 27-30, wherein luminance is inherent to environment 
maps and texture maps.). 

Regarding claim 33, as it depends from claim 32, Szeliski teaches a 
system wherein the input images comprise two-dimensional images (See 
Szeliski: Figs. 3-4; Col. 9, Lns. 56-58, wherein a camera 310 having its 
optical center fixed at point C (FIG. 3) captures a sequence of 2D still 
images Io, Ii, h, h...). 
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Regarding claim 34, as it depends from claim 32, Szeliski teaches the 
input images comprise three-dimensional images including geometry 
information (See Szeliski: Figs. 3-4 and 6; Col. 9, Lns. 56-62, wherein the 
camera captures a sequence of 2D still images (Io, Ii, h, h.) as it pans, 
the center rays of these images being focused on 3D points (Po, Pi, P2, 
P3...) at a focal length f from the optical center point C. The points Pi are 
defined relative to a fixed 3D world coordinate system (P x , P y , P z ). Since 
the three-dimensional images correspond to the two-dimensional images 
which include depth information in the form of focal length. The 
geometry information is the inverted V shape shown in both figures 4 
and 6.). 

Regarding claim 35, as it depends from claim 32, Szeliski teaches 
aligning the image panoramas according to instructions received from a 
user (See Szeliski: Col. 27, Lns. 64-66). 

Regarding claims 40 and 41, as they depend from claims 22 and 38, 
respectively, Szeliski teaches the object is a room and the at least one 
boundary of the object is the intersection of a wall of the room with the 
floor (See Szeliski: Col. 27, Lns. 64-67, wherein the model can be a cube 
with six separate texture maps for each surface. Using an appropriate 
environment map would cause the top surface to be a ceiling, the bottom 
surface to be a floor, etc.). 
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B. Claims 11, 13-21, 23, and 36-37 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over the combination of Szeliski and Seago, and in further view of 
Blank (US Patent Number 5,469,536). 

Regarding independent claim 1 1 , Szeliski teaches a computerized method 
of interactively editing objects in a panoramic image, the method 
comprising: 

• Receiving an image panorama representing a visual scene (See 
Szeliski: Col. 27, Lns. 34-37, "Once a complete panoramic mosaic has 
been constructed, it is necessary to convert the set of input images 
and associated transforms into one or more images which can be 
quickly rendered or viewed."), 

o The image panorama having an object (See Szeliski: Fig. 6, Item 
610; Col. 11, Lns. 16-21) and a point source (See Szeliski: Figs. 3-4 
and 6; Col. 9, Lns. 56-62, wherein the camera captures a sequence 
of 2D still images (Io, Ii, h, h.) as it pans, the center rays of these 
images being focused on 3D points (Po, Pi, P2, P3...) at a focal length 
/from the optical center point C"), 

o The object occupying a field of view of more than 180 degrees in 
the panorama (See Szeliski: Col. 5, Lns. 7-9, "By taking as many 
images as needed, image mosaics can be constructed which cover 
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as large a field of view as desired, e.g., a complete environment 
map."); 

Szeliski does not expressly suggest: 

• Creating a three dimensional model of the visual scene using features 
of the visual scene and the point source, wherein: 

o Creating a three dimensional model includes identifying at least 

one boundary of the object and 
o Using the identified boundary to associate geometry information 

with the object, 

■ The geometry information comprising 3-D coordinates 
describing the position and orientation of the object boundary 
in a reference coordinate system. 

Seago is cited for teaching: 

• Creating a three dimensional model of the visual scene (See Seago: 
Fig. 2; Col. 4, Lns. 23-25, "FIG. 2 illustrates the process by which the 
image converting system 20 generates a three-dimensional model or 
object from a single two-dimensional image.") using features of the 
visual scene and the point source (See Seago: Fig. 2, Item 40; Col. 4, 
Lns. 34-37, "First, at block 40, a digitized two-dimensional 
perspective image of one or more objects is retrieved from memory 32 
or a digital image input device 28 and displayed on the display device 
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24," wherein the features of the visual scene are the one or more 
objects and the point source is indicated by the perspective of the 
image.), wherein: 

o Creating a three dimensional model includes identifying at least 
one boundary of the object (See Seago: Col. 4, Line 65 - Col. 5, Line 
1, respectively, "Line creation can be performed automatically by 
image analysis software, which determines edges of objects and 
creates lines overlapping the determined edges.") and 

o Using the identified boundary to associate geometry information 
with the object (See Seago: Fig. 2, Item 50; Col. 5, Lns. 35-38, 
"Next, at blocks 50, shapes or polygons that define the selected 
object's sides, and plane indexes are determined based on user 
designated vertices or vanishing lines at significant features of the 
selected object."), 

■ The geometry information comprising 3-D coordinates 
describing the position and orientation of the object boundary 
in a reference coordinate system (See Seago: Fig. 2, Item 54; 
Col. 5, Lns. 50-55, "Once all the polygons and plane indexes of 
a selected object have been determined, a three-dimensional 
object oriented within the selected object's three-dimensional 
coordinate space is determined."). 
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It would have been obvious to one of ordinary skill in the art, at the time 
the invention was made, to have used the three-dimensional object 
editing abilities of Seago, to modify the three-dimensional environment 
map, as taught by Error! Reference source not found., because Seago 
is in the same field of endeavor of generating three-dimensional 
computer graphics using image capture devices; and Seago expressly 
suggests that this method of object extraction produces accurate three- 
dimensional objects more efficiently than conventional systems that try 
to extract the three-dimensional objects using analytical mathematical 
interpretations and orthogonal image analysis (See Seago: Col. 11, Lns. 
49-55). 

The combination of Szeliski and Seago does not expressly suggest: 

• Receiving an edit to the object in the panorama; 

• Transforming the edit relative to a viewpoint defined by the point 
source; and 

• Projecting the transformed edit onto the object. 

Blank is cited for teaching trimming, which allows the user to trim off 
any undesired edges of an object to reveal the background below (See 
Blank: Col. 47, Lns. 11-20). It would have been obvious to one of 
ordinary skill in the art, at the time the invention was made, to have 
allowed modification of the colors within an image, as taught by Blank, 
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to manipulate the colors within the images used to create the panoramic 
image mosaics, as taught by Szeliski, as modified by Seago, because it 
would enable the user to quickly and efficiently modify or enhance the 
appearance of an image to desired goal (See Blank: Col. 6, Lns. 23-28). 

Independent claim 36 differs from independent claim 1 1 primarily in that 
claim 36 is directed toward the system for implementing the 
computerized method of claim 1 1. Claim 36 also differs in that it requires 
creating a three dimensional model includes identifying at least one 
boundary of the object and one or more interactive editing tools for 
providing an edit to the selected object. Seago teaches, "FIG. 2 illustrates 
the process by which the image converting system 20 generates a three- 
dimensional model or object from a single two-dimensional image (See 
Seago: Fig. 2; Col. 4, Lns. 23-25)," wherein "... at block 42, an object 
contained within the digital image is selected for conversion into a three- 
dimensional object, and the selected object's orientation, or natural 
coordinate system, is approximately determined (See Seago: Fig. 2, Item 
42; Col. 4, Lns. 46-49)," which corresponds to creating a three 
dimensional model includes identifying at least one boundary of the 
object. Blank is cited for teaching function calls, which correspond to the 
interactive editing tools for editing the selected object (See Blank: Col. 21, 
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Lns. 18-23). Thus, the rationale of independent claim 1 1 is applied to 
independent claim 36. 

Regarding claim 13, as it depends from claim 11, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
receiving an edit to color information associated with the object (See 
Blank: Col. 6, Lns. 29-47). 

Regarding claim 14, as it depends from claim 11, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
receiving an edit to alpha information associated with the object (See 
Blank: Fig. 11; Col. 20, Lns. 61-66). 

Regarding claim 15, as it depends from claim 11, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
receiving an edit to depth information associated with the object (See 
Blank: Col. 13, Lns. 8-16). 

Regarding claim 16, as it depends from claim 11, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
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receiving an edit to geometry information associated with the object (See 
Blank: Col. 47, Lns. 11-20). 

Regarding claim 17, as it depends from claim 11, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
providing a user with an interactive drawing tool that specifies edits for 
the object and receiving the edits made by the user using the interactive 
drawing tool (See Blank: Col. 21, Lns. 18-23). 

Regarding claims 18 and 37, as they depend from claims 17 and 36, 
respectively, the rationale of claim 11 is incorporated herein. The 
combination of Szeliski, Seago, and Blank substantially teach the 
invention as claimed. Specifically, Blank teaches the interactive drawing 
tool is a depth chisel tool (See Blank: Col. 47, Lns. 1 1-20). 

Regarding claim 19, as it depends from claim 17, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
the interactive drawing tool specifies a selected value for depth for the 
object (See Blank: Col. 22, Lns. 35-62). 

Regarding claim 20, as it depends from claim 17, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
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substantially teach the invention as claimed. Specifically, Blank teaches 
the interactive drawing tool incrementally adds to the depth for the object 
(See Blank: Col. 34, Lns. 8-11). 

Regarding claim 21, as it depends from claim 17, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
the interactive drawing tool incrementally subtracts from the depth for 
the object (See Blank: Col. 34, Lns. 22-26). 

The rationale of claim 17 is applied to claim 23. 

C. Claims 1-3, 5-6, 8-10 and 39 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Szeliski in view of Luken (US Patent Number 5,923,334), and 
in further view of Seago. 

Regarding independent claim 1, Szeliski teaches a computerized method 
for creating a three dimensional model from image panoramas, the 
method comprising: 

• Receiving a plurality of image panoramas representing a visual scene 
(See Szeliski: Col. 27, Lns. 34-37, "Once a complete panoramic mosaic 
has been constructed, it is necessary to convert the set of input 
images and associated transforms into one or more images which can 
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be quickly rendered or viewed.") and having an object (See Szeliski: 
Fig. 6, Item 610; Col. 11, Lns. 16-21), 

o The object occupying a field of view of more than 180 degrees in 
the panoramas (See Szeliski: Col. 5, Lns. 7-9, "By taking as many 
images as needed, image mosaics can be constructed which cover 
as large a field of view as desired, e.g., a complete environment 
map."); 

• Determining a directional vector for the image panorama, the 
directional vector indicating an orientation of the visual scene with 
respect to a reference coordinate system (See Szeliski: Fig. 15; Col. 22, 
Lns. 4-31); 

• Transforming the image panoramas such that the directional vectors 
are substantially aligned relative to the reference coordinate system 
(See Szeliski: Fig. 15; Col. 22, Lns. 2-4, "FIG. 15 shows the 
adjustment of the bundle of rays Xjk so that they converge to x/."); 

• Aligning the transformed image panoramas to each other (See 
Szeliski: Fig. 17; Col. 23, Lns. 10-12, wherein minimizing the 
differences between ray directions aligns the images.). 

Szeliski does not expressly suggest determining a directional vector for 
each image panorama. Luken is cited for teaching eight direction vectors 
D0-D7 associated with six rectangular images mapped to the inside of an 
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octahedron (See Luken: Figs. 7-10, 14 and 17; Col. 7, Lns. 28-36, 
wherein it is determined which of the six rectangular images is 
intersected by one of the eight direction vectors.). It would have been 
obvious to one of ordinary skill in the art, at the time the invention was 
made, to have used direction vectors for each image within an 
environment map that is mapped to the sides of an octahedron, as 
taught by Luken, with the three-dimensional model environment map of 
the visual scene, as taught by Szeliski, because Luken: (1) is directed to 
the same problem of using polyhedral environment maps to create and 
view three dimensional images from data representing multiple views of a 
scene; (2) is in the same field of endeavor of image processing systems; 
and (3) Luken expressly suggests that the direction vectors provide an 
efficient system for generating and viewing three-dimensional panoramic 
images based environment maps, and offer an improved level of 
interactive graphical feedback (See Luken: Col. 3, Lns. 5-8). 

The combination of Szeliski and Luken does not expressly suggest: 

• Creating a three dimensional model of the visual scene from the 

transformed and aligned image panoramas using the reference 

coordinate system, wherein 

o Creating a three dimensional model includes identifying at least 
one boundary of the object and 
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o Using the identified boundary to associating associate geometry 
information with the object, 

■ The geometry information comprising 3-D coordinates 
describing the position and orientation of the object boundary 
in the reference coordinate system. 

Seago is cited for teaching: 

• Creating a three dimensional model of the visual scene (See Seago: 
Fig. 2; Col. 4, Lns. 23-25, "FIG. 2 illustrates the process by which the 
image converting system 20 generates a three-dimensional model or 
object from a single two-dimensional image.") from the transformed 
and aligned image panoramas using the reference coordinate system 
(See Seago: Fig. 2, Item 42; Col. 4, Lns. 46-49, "Next, at block 42, an 
object contained within the digital image is selected for conversion 
into a three-dimensional object, and the selected object's orientation, 
or natural coordinate system, is approximately determined," wherein 
the reference coordinate system is the natural coordinate system.), 
wherein 

o Creating a three dimensional model includes identifying at least 
one boundary of the object (See Seago: Col. 4, Line 65 - Col. 5, Line 
1, respectively, "Line creation can be performed automatically by 
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image analysis software, which determines edges of objects and 
creates lines overlapping the determined edges.") and 
o Using the identified boundary to associating associate geometry 
information with the object (See Seago: Fig. 2, Item 50; Col. 5, Lns. 
35-38, "Next, at blocks 50, shapes or polygons that define the 
selected object's sides, and plane indexes are determined based on 
user designated vertices or vanishing lines at significant features of 
the selected object."), 

■ The geometry information comprising 3-D coordinates 
describing the position and orientation of the object boundary 
in the reference coordinate system (See Seago: Fig. 2, Item 54; 
Col. 5, Lns. 50-55, "Once all the polygons and plane indexes of 
a selected object have been determined, a three-dimensional 
object oriented within the selected object's three-dimensional 
coordinate space is determined."). 

It would have been obvious to one of ordinary skill in the art, at the time 
the invention was made, to have used the three-dimensional object 
editing abilities of Seago, to modify the three-dimensional environment 
map, as taught by Error! Reference source not found., as modified by 
Luken, because Seago is in the same field of endeavor of generating 
three-dimensional computer graphics using image capture devices; and 
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Seago expressly suggests that this method of object extraction produces 
accurate three-dimensional objects more efficiently than conventional 
systems that try to extract the three-dimensional objects using analytical 
mathematical interpretations and orthogonal image analysis (See Seago: 
Col. 11, Lns. 49-55). 

Regarding claim 2, as it depends from claim 1, Szeliski teaches the 
directional vector is determined based, at least in part, on instructions 
identifying elements of the image panoramas received from a user (See 
Szeliski: Col. 8, Lns. 30-32 and Col. 27, Lns. 64-66). 

Regarding claim 3, as it depends from claim 2, Szeliski teaches the 
instructions from the user identify two or more substantially parallel 
features in the image panoramas (See Szeliski: Col. 20, Line 64 - Col. 21, 
Line 6 and Lines 25-34). 

Regarding claim 5, as it depends from claim 2, Szeliski teaches the 
instructions from the user identify a horizon line of at least one image 
panorama (See Szeliski: Fig 4: Col. 9, Lns. 54-62). 

Regarding claim 6, as it depends from claim 2, Szeliski teaches the 
instructions comprise the identification of two or more areas of the image 
panoramas, each area containing one or more elements and further 
comprising automatically identifying the two elements contained in the 
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two or more areas (See Szeliski: Fig. 6; Col. 20, Line 49 - Col. 21, Line 
24, wherein a feature-based point correspondence is established between 
a pair of images by dividing each image into patches and identifying 
prospective "feature" points within the patches.). 

Regarding claim 8, as it depends from claim 1, the rationale of claim 1 is 
incorporated herein. The combination of Szeliski, Luken, and Seago 
substantially teach the invention as claimed. Specifically, Luken teaches 
the image panoramas are aligned relative to the reference coordinate 
system such that the directional vector of each panorama is at least 
substantially parallel to one axis of the reference coordinate system (See 
Luken: Fig. 7, Item 707; Col. 6, Line 40— Col. 7, Line 36). 

Regarding claim 9, as it depends from claim 1, the rationale of claim 1 is 
incorporated herein. The combination of Szeliski, Luken, and Seago 
substantially teach the invention as claimed. Specifically, Luken teaches 
the image panoramas are aligned relative to the reference coordinate 
system such that the directional vector of each panorama is at least 
substantially orthogonal to one axis of the reference coordinate system 
(See Luken: Fig. 6A and 7, Item 707; Col. 6, Line 40— Col. 7, Line 36, 
wherein since the six rectangular images are axis aligned, then a 
directional vector that is parallel to one axis must be perpendicular to 
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the other two spatial axes. In other words, in order for a directional 
vector to intersect one of the rectangular images, that vector must be 
substantially parallel to one axis, which requires it to be substantially 
perpendicular to the others.). 

Regarding claim 10, as it depends from claim 1, Szeliski teaches the 
image panoramas are aligned according to instructions received from a 
user (See Szeliski: Col. 27, Lns. 64-66, wherein the user aligns the 
image panoramas into texture space.). 

Regarding claim 39, as it depends from claim 1, Szeliski teaches the 
object is a room and the at least one boundary of the object is the 
intersection of a wall of the room with the floor (See Szeliski: Col. 27, 
Lns. 64-67, wherein the model can be a cube with six separate texture 
maps for each surface. Using an appropriate environment map would 
cause the top surface to be a ceiling, the bottom surface to be a floor, 
etc.) 

D. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over the 
combination of Szeliski, Luken, and Seago, and in further view of Blank. 

Regarding claim 7, as it depends from claim 6, the combination of 
Szeliski, Luken and Seago does not expressly suggest using edge 
detection to automatically identify the two elements. Nevertheless, Blank 
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teaches detecting the edges of an object and separates portions of the 
image that are outside the edge of the object (i.e., the background 
component) from portions of the image that are inside the edge (See 
Blank: Col. 4, Lns. 17-21). The two elements are therefore identified as 
those elements within the edge, and those outside the edge. It would 
have been obvious to one of ordinary skill in the art, at the time the 
invention was made, to have used the edge detection methods, as taught 
by Blank, as an alternative to patch-based division, as taught by Szeliski, 
as modified by Luken, because it is an effective way to divide the image 
into smaller portions to conquer aligning all aspects of an image. 

Conclusion 

Any inquiry concerning this communication or earlier communications 
from the examiner should be directed to Crystal Murdoch whose 
telephone number is (571) 270-1043. The examiner can normally be 
reached on Mon. - Fri. 10:00 am to 6:30 pm. If attempts to reach the 
examiner by telephone are unsuccessful, the examiner's supervisor, Kee 
Tung can be reached on (571) 272-7794. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273- 
8300. 
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Information regarding the status of an application may be obtained from 
the Patent Application Information Retrieval (PAIR) system. Status 
information for published applications may be obtained from either 
Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more 
information about the PAIR system, see http://pair-direct.uspto.gov. 
Should you have questions on access to the Private PAIR system, contact 
the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you 
would like assistance from a USPTO Customer Service Representative or 
access to the automated information system, call 800-786-9199 (IN USA 
OR CANADA) or 571-272-1000. 
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